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Abstract. Monotone triangles are certain triangular arrays of integers, which cor- 
respond to n X n alternating sign matrices when prescribing (1,2,..., n) as bottom 
row of the monotone triangle. In this article we define halved monotone triangles, a 
specialization of which correspond to vertically symmetric alternating sign matrices. 
We derive an operator formula for the number of halved monotone triangles with 
prescribed bottom row which is analogous to our operator formula for the number of 
ordinary monotone triangles [5]. 



1. Introduction 

Alternating sign matrices and equivalent objects such as fully packed loop config- 
urations, the six vertex model and monotone triangles are nowadays a rich source 
for intriguing problems on which combinatorialists can test their various enumeration 
methods. This article is another contribution in this respect. 

In [2] we gave a formula for the number of monotone triangles with prescribed bot- 
tom row. Strikingly this formula involves shift operators which are applied to a simple 
multivariate polynomial. It is an example of a new type of enumeration formula com- 
binatorialists can possibly make use of when answering their enumeration problems. 
Subsequently, our formula enabled us to give a new proof of the refined alternating 
sign matrix theorem [3], which was first proved by Zeilberger [12]. Here, we present a 
second example of such an operator formula. This new formula gives the number of 
halved monotone triangles with prescribed bottom row, a notion to be defined below. 

To keep the treatment self-contained, we recall the basic definitions. An alternating 
sign matrix is a square matrix with Os, Is and —Is as entries such that the row- and 
columnsums are 1 and the non-zero entries of each row and of each column alternate 
in sign. Thus, 

/O 1 \ 
1 0-1 10 
1-1 1 0-11 
1-110 
1-1 1-1 10 
1-110 
\0 1 00/ 
1 
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is an alternating sign matrix. The fascinating story of alternating sign matrices pQ 
began when combinatorialists where confronted with a conjecture by Mills, Robbins 
and Rumsey ^ [7] , which states that the number of n x n alternating sign matrices is 
given by the following simple formula 

(3j + l)! 



n 



,=0 ^^ + ^^' 

For a long time no one could explain this, until finally Zeilberger [TT] came up with 
the first proof. Soon after another, shorter, proof was given by Kuperberg [Jj. See also 
[3], where we have recently presented a new proof of this result. 
A monotone triangle is a triangular array (aij)i<j<i<n of integers. 
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is a monotone triangle. It corresponds to the alternating sign matrix above: in the 
matrix, replace every entry with the sum of entries in the same column above, the 
entry itself included. The result is a 0-1-matrix with one 1 in the first row, two Is 
in the second row etc. If one records the columns of the Is rowwise, one obtains the 
corresponding monotone triangle. It is not hard to see that this establishes a bijection 
between monotone triangles with bottom row (1, 2, . . . , n) and n x n alternating sign 
matrices. 

Observe that the alternating sign matrix given above is symmetric with respect to 
the vertical symmetry axis. This is not the case for all alternating sign matrices. In 
fact there only exist vertically symmetric alternating sign matrices of odd size. (This 
follows from the fact that an alternating sign matrix has always a unique 1 in its top 
row.) Kuperberg [5] showed that the number of vertically symmetric {2n + 1) x [2n-\-l) 
alternating sign matrices is given by 

n! (6j-2)! 



(2n)!2'^ (2n + 2j - 1)!' 
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(This formula was conjectured by Robbins [ID]-) The symmetry of vertically symmetric 
alternating sign matrices translates into a symmetry of the corresponding monotone 
triangle: the replacement of every entries e by 2n + 2 — e and the subsequent reflection 
along the vertical symmetry axis leaves the monotone triangle invariant. Thus, in case 
of a vertically symmetric alternating sign matrix, it suffices to "store" (a bit less than) 
half of the monotone triangle. In our example, this is the following array. 

2 

1 

1 3 
1 2 
12 3 
12 3 



Note that the middle column of a monotone triangle corresponding to a {2n + 1) x 
(2n + 1) vertically symmetric alternating sign matrix consists solely of {n + l)s and, 
consequently, we do not have to store it. 

These considerations led us to the following definition. A halved monotone triangle 
is a triangular array {0'i,j)i<i<n,i<j<\^] integers, 

^1,1 

02,1 

0-3,1 0-3,2 
04,1 04,2 
05,1 05,2 05,3 

06,1 06,2 06,3 



which is monotone increasing in northeast and southeast direction and strictly increas- 
ing along rows, that is Oi+ij < aij, aij < Oj+ij+i and aij < ajj+i for all The bijec- 
tion sketched above shows that halved monotone triangles (oi,j)i<i<2n,i<j<r|] ^^^^ 
tom row (1,2,..., n) such that no entry is greater than n correspond to (2n+l) x [2n+l) 
vertically symmetric alternating sign matrices. We are ready to state the main result 
of the paper. 

Theorem 1. The number of halved monotone triangles with n rows, where no entry 
exceeds x and with bottom row {ki, . . . , k^n/2'\), ki < k2 < ■ ■ ■ < k^n/2\ < is equal to 



n + K - id) + - id) 

J<p<q<(n+l)/2 

n 



l<i<j<{n+l)/2 \J )\J I 
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if n is odd and equal to 

n ^>^r. + - id) + ^>^^ - id) 

^l<p<g<n/2 

n/2 

n{kj — ki) {2x + 2 — ki — kj) -p-r x + 1 — ki 

n is even, where E^p^x) = p{x + 1) denotes the shift operator. In this formula, 
the product of operators is the composition, and, since the shift operators with respect 
to different variables commute, we do not have to specify the order in which they are 
applied. 

We think that the following phenomenon is interesting, since it is in analogy to 
the situation for ordinary monotone triangles, see [2j. If we consider "halved mono- 
tone triangles" which do not necessarily have strict monotony along rows (the weak 
monotony follows from the other conditions), the enumeration problem is significantly 
easier. These objects are equivalent to shifted plane partitions of trapezoidal shape 
with prescribed diagonal and were enumerated by Proctor [8i Prop. 4.1]. The number 
of these halved triangles with n rows, where no entry exceeds x and with bottom row 
(/ci, . . . , fc[„/2]) is equal to 

{kj — ki + j — i) (2x + 2 + n — i — j — ki — kj) 



n 

l<i<j<(n+l)/2 

if n is odd and equal to 

n/2 

n{kj — ki + j — i) (2s + 2 — i— j + n — ki — kj) -p-i- x + 1 — i + n/2 ~ ki 

l<i<j<n/2 J\J ' / i^i 

if n is even. Let P{n,x; ki, . . . , k^n/2]) denote the number of these objects. Then, by 
Theorem [1], the number of halved monotone triangles with n rows, where no entry 
exceeds x and with bottom row (fci, . . . , fc|-n/2]) is given by 

n {Ek, + E,^ - E,^E,^){E,^ + E-' - Eu^E-^) /?(n, x; fc^, . . . , /.p./^i). 
,i<P<g<r"/2l / 

This happens to be in perfect analogy to the situation for ordinary monotone trian- 
gles: an enumeration formula for the objects with strict monotony along rows can be 
obtained by applying a product of simple operators to an enumeration formula for the 
corresponding objects with weak monotony along rows, the latter of which is a simple 
product formula. 

The paper is organized as follows. We prove Theorem [1] in Sections [2] - [5l Our 
strategy is to first show the polynomiality of the formula, then compute its degree, and 
finally derive enough properties that characterize the polynomial. To be more precise, 
in Section [21 we introduce the recursion underlying our enumeration formula for halved 
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monotone triangles and show the polynomiahty of the formula. In Section [SI we define 
an operator, which is closely related to the recursion and prove a number of lemmas 
on it. In Section m we list and derive the properties that characterize the enumeration 
polynomial, and, in Section 0, we finally use these properties to prove Theorem [H 
In Section [6], we use our operator formulas to derive a generating function for halved 
monotone triangles as well as a generating function for ordinary monotone triangles. 



2. A RECURSION AND THE POLYNOMIALITY OF THE ENUMERATION FORMULA 

For n > 1 and ki < k2 < ■ ■ ■ < k^^-^ < x, let 'y{n,x;ki, . . . ,k^:i-^) denote the 
quantity we want to compute, i.e. the number of halved monotone triangles with n 
rows, where the bottom row is {ki, k2, ■ ■ ■ , kt^n,-^ ) and all entries are no greater than 
X. We define a summation operator for functions /(/i, . . . ,/m-i), where m > 2 and 
(/i, . . . , Im-i) G Z™~^, as follows. For given {ki, . . . , km) G we have 

{h,...,lm-l) (!i,....i„_i)6Z'"-l, 

ki<li<k2<...<k^_l<l^_l<km.,li7^li+l 

i.e. we sum over all strictly increasing sequences (/i, . . . , Im-i) such that k^ <li < fcj+i 
for all i. This operator is well-defined for all strictly increasing sequences (fci, . . . , km) G 
Z™. If we define 7(0, x; — ) = 1 we have the following recursions. If n is even then 

{fci,...,A:„/2.a:) 

7(n,x;fci,...,/c„/2) = ^ 7(n - 1, x; /i, ^2, • • • , 

{h,---,ln/2) 

and if n is odd then 

7(n,a;;/ci,...,A;(„+i)/2) = ^ 7(n - 1, x; /i, . . . , /(„_i)/2). 

We want to extend the interpretation of 7(n, x; fci, . . . , ) to arbitrary (fci, . . . , ) G 

1 _ Yoi this purpose, it suffices to extend the definition of (12.11) to arbitrary (fci, . . . , km) G 
Z™ and then use the recursions to define the generalization of 7. We use induction 
with respect to m. For m = 2, let 

^ a(/i) := ^ a(/i), 

ih) h=ki 
b a-1 

where here and in the following ^ f{i) = — ^ f{i) if a > b. (Note that this implies 

i=a i=b+l 

a-1 y 

^ f{i) = 0. Moreover, ^ p{x) will be a polynomial in y if j9(x) is a polynomial in x.) 

i=a x=0 
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If m > 2 we define 

a(Zi, . . . , Im-l) '■— 

{kl,...,kjn—l) km (kl,...,km—2,km—l — i) 

• • • , ^m-l) + ^(^l , • • • , ^i^m-l) • 

(/l,...,irn-2) 'm-l=fcm-l + l (il,...,/„i_2) 

Now it is also obvious that j{n, x;ki,..., k^n/2\) is a polynomial in {ki, . . . , k^n/2], 
for fixed n. 

This recursion can be used to compute 7(n, x;ki,..., k^n/2\) for small values of n. 
For n = 1, 2, 3, 4, 5 you find the results below. 



1, X - A;i + 1, -(2x + 2 - /ci - k2){k2 -ki + 1), -{x-k2 + l){-kl + Sxkj + 6kl - 2x^ki 
+ klki - IQxki - 2xk2ki - 2k2ki -llki + 2x^ - xkl - 2kl + 7x + 2x^k2 + ^xk2 + 4^2), 

^{klk\ - kjkf + 2xk{ - 2xk2k\ - 3k2kl + 2xk^ + k^k\ + 2k\ - Sx^kf - ixk^kf 
48 

- 10^2^1 + Axkjkf + lOklkf - 28xkl + 8x^k2kf + 32xk2kf + 30^2^;^ - Sx^k^kf - 2Axk^kl 
-10k3kl-20kl-k^kl + k^kl+8x^kl+4xklkl+6klkl-4xklkl-2klkl+60x^kl + 12xklkf 
+2Aklkl-2Axklkj-36klkl + 122xkl-8x^k2kl-60x'^k2kl-138xk2kl-99k2kl+8x^k3kl 
+60x'^k3kl+102xk^kl+37kskl+70kl+2xk%+5k%-2xktki-5ktki-2Ax^ki-8x^k:lki 
-32xklki-30klki+8x'^klki+24:xklki+10klki-124:X^ki+8x^klki+36x'^klki+42xklki 
+29klki-Sx^klki-12x^klki+4:2xklki+31klki-152xki+16x^k2ki+96x'^k2ki+UQxk2ki 
+ A8k2ki - 32x^kzki - lUx'^k^ki - l3Qxk^ki - 36kski - 52ki - 4.xk2 - 6A;2 -klkl + 2xkl 
+2xk2kl+3k2kl+^kl+l&x'^kl+A&xkl+28kl-8x'^kl+4.xklkl+2klkl-2Qxkl-8x^k2kl 
-l&xk2kl - 6k2kl - 8kl - IGx^kj - 72x^kl -100xkl-66kl+k^kl+8x^kl-4:xklkl-6klkl 
+12x^kl+12xklkl+12klkl-22xkl+8x^k2kl+12x^k2kl-6xk2kl+15k2kl-Akl+32x'^k2 
+ 96xk2 + Uk2 - 2xklkz -k^k^^ 2Ax^k3 + 8x^klkz + IQxklk^ + Qklk^ + 92x^^3 - 8x^klkz 
-3Qx^klkz-^2xklk^-l3klkz + hQxk^ + lQx^k2k^ + 48a;^ k2 k^ - Axk2 k^ -12k2 k^ + 8 ^3 ) 

This data suggests that the degree of '~f{n, x;ki,..., k^n/2) in ki is always n — 1. That 
this is indeed the case will be shown in the following section. However, this comes by 
surprise because of the following: suppose that a{li, . . . , Im-i) is a polynomial of degree 
R in every k. Then the degree of 

a(/i, . . . , 

{ll,...,lm-l) 
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m—l 

in ki could be as high as 2R + 2 (e.g. a{li, . . . ,lrn-i) = 11 ^f )- This estimation 

1=1 

provides (by induction with respect to n) a bound of 2""''^ — 2 for the degree of 
7(n,x; ki,..., A;p„/2]) in ki. 

3. An OPERATOR RELATED TO THE RECURSION 

Most of the definitions and lemmas in this section are taken from [2]. The shift 
operator Ex is defined as Exp{x) = p{x + 1) and the difference operator is defined 
as Ex — id. The swapping operator Sx,y is defined as Sx,yf{x,y) = f{y,x). 

Note that the shift operator Ex is invertible as an operator over C[x], whereas the 
difference operator A^. is not, since it decreases the degree of a polynomial. In the 
following, we will consider rational functions in shift operators and thus we need a 
lemma in order to show that the inverses of our denominators exist. (The lemma is a 
generalization of [21 Lemma 1].) For the statement of the lemma we need the following 
observation. Let 

p{Xi, . . . , Xn) = ^ aii,...,i„Xl^ ■ ■ ■ X^" 
(n,...,i„)e(Z>o)" 

be a formal power series in {Xi, . . . , X„) over C and G{ki, . . . , kn) be a polynomial in 
(fci, . . . , kn) over C. We define 

p{Ak„. . . , AkJG{ku ...,kn):= J2 • ■ ■ A^^Giki, . . . , A;„). 

(ji,...,i„)e(z>o)" 

This is a finite sum and thus well-defined since A^^^G(A;i, . . . , kn) = if deg;;.. G{ki, . . . , kn) = 
d. 

Lemma 1. Let p{Xi, . . . , X„) be a formal power series in (Xi, . . . , X„) over C with 
non-zero constant term. Then p{Ak^, . . . , A^^) is invertible as an operator overC[ki, . . . , kn], 
i.e. there exists a formal power series q{Xi, . . . , X„) with 

p{Ak^, . . . , AfcJg(Afc,, . . . , AkjF{ki, ...,kn) 

= q{Ak,, . . . , AkJp{Ak„ . . . , Ak„)F{ki, ...,kn)= F{ki, ...,kn) 

for all polynomials F{ki, . . . , kn). Moreover 

degfc,^,...,fc,^ G{ki, ...,kn) = deg^^^^ p{Ak„ Ak„)G{ki, ...,kn) = 

degfc^^,...,fc,^ q{Ak„ AkJG{ki, . . . , fc„) 

for all {ii, Z2, . . . , im) with 1 < ii < i2 < . . . < im ^ n, where degj..^ ^. G{ki, . . . , 
denotes the degree of G{ki, . . . , as a polynomial in {ki^, . . . , fcj^). 

Proof. The assertion follows from the fact that p(Xi, . . . ,X„) is invertible in the 
(commutative) algebra of formal power series over C if (and only if) j5(Xi, . . . ,X„) 
has a non-zero constant term. This is because p(Xi, . . . , X„)g(Xi, . . . , X„) = 1 is 
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equivalent to ao,...,o^o,...,o = 1 and 

(ii,...,i„),(ji,...j„)e(z>0)" 

(il,...,i„) + (jl,... J„) = (rj,...,r„) 

for (ri, . . . , r„) 7^ (0, . . . , 0), where 

piX,,...,X^)= Yl <^n,-,3.Xt ■ ■ ■ Xt ■ 
(ii,...j„)e{z>0)" 

and 

(ii,...,i„)e{z>0)" 

By assumptfon ao,...,o 7^ and, consequently, the equatfons allow us to determine the 
coefficients &ri,...,r„ by induction with respect to ri + . . . + r„. The assertion about the 
degree follows from the fact that 

degfc.^,...,fc^^ p(Afc,, . . . , AfeJG'(A;i, . . . , < deg^^^^ ^(fci, . . . , fc„) 
= degfc^^^_...^fc^^ g(Afc,, . . . , AfcJp(Afe,, . . . , AfcJG'(/ci, . . . , A;„) 

< degfc^^^^...^fc^^ p(Afc,, . . . , AfcJG(fci, . . . , □ 

We define Vx,y = (id +-Ej,Aa;) = + A^^A^^. In the following lemma we will see why 
this operator is of significance for the recursion underlying 

ln/2-])- (The 

lemma is equivalent to p], Lemma 2].) It will be used for showing that the degree of 
'j{n,x] ki, . . . , fc[n/2]) is no greater than — 1 in every fcj. 

Lemma 2. Let a{x, y) be a polynomial in x and y which is of degree no greater than R 
in each of x andy. Moreover, assume that {id +Sx,y)Vri.^ya{x,y) is of degree no greater 
than R as a polynomial in x and y, i.e. a linear combination of monomials x"^y^ with 
m + n < R. Then 

a(x,?/) = ^ ^ a(x,?/) - a(A;2, A:2) (3.1) 

{x,y) x=ki y=k2 

is of degree no greater than R + 2 in ^2. Moreover, if [id +Sx^y)Vx^ya{x,y) = then the 
degree of fl3.1l) in ^2 is no greater than R + 1. 

Proof. First note that, by Lemma[T], V^^y + Vy^x = 2 id +A^+Aj^ + 2A^.Aj^ is invertible. 
Thus 

(id+gx,,) .. + (id-g..,) = id, (3.2) 

^x,y 1 Vy,x ^x,y i Vy,x 

since V^^y + Vy^^ and S^^y commute. Moreover, 

(id+g^J a{x,y) = ^ {id +Sx,y)Vx,ya{x,y). 

^x,y ~r '/y,x ^x,y ~r '/y,x 
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By Lemma [T], the degree of this expression in x and y is equal to the degree of 
(id +Sx,y)Vx^ya{x, y) in x and y and the expression vanishes if and only if (id +Sx,y)Vx^ya{x, 
vanishes. Thus, by (I3.2p . it suffices to show that the degree of 



{ki,k2,kz) 



y,x 



{id -S^^y)a{x,y) 



in ^2 is no greater than R + 1. Once more the degree estimation from Lemma [T] 
implies that this can be reduced to showing the following. If we define b{x,y) = 

(k\,k2M) 

V^^a;(id —Sx^y) (^) (^) then the degree of ^ 6(a;, y) in /c2 is no greater than max(p, g) + 



1. In order to do so, observe that 



h{x,y) = Vy^x{id-Sx^y)(^ 



qj \p 



+ 



x + 1 
P 



y 

q-l 



x + 1 



y 

p — 1 



Therefore, and by the summation formula 

b 



X + 1 
n + 1 



X 



n 



6 + 1 



we have 



k2 fcs 



2^ 2^ K^^y) - b{h,k2) 

x=ki y=k2 



+ 



k2 + l 
q + l 
k2 + 2 
p + l 
k2 + 2 
q + l 



k2 + l 
p + l 

ki 
q+l 
ki + 1 
p+l 
ki + 1 
q + l 



n 



ki 
p+l 

ks + l 
p+l 
ks + l 
Q 

ks + l 
P 

k2 + l 
P 



a 

n + 1 



ks + l 
q + l 

k2 
p+l 
k2 

1 
k2 

P 



k2 
q + l 



k2 
q-l 



+ 



k2 + l 



k2 
p — 1 



If we repeatedly apply the identity 

n 
k 



n — 1 
k 



72 — 1 
k-1 



to this expression, we finally see that this is a polynomial in k2 of degree no greater 
than max(p, g) + 1. □ 

In order to use this lemma to compute the degree of ■y{n, x;ki,..., k^n/2]) in every 
ki, we need to show that {id +Sk,,k,+i)Vk^^k,+il{n, x; ki, . . . , k^n/2\) = for all i. This 
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will be a consequence of the following lemma, which implies that 

{h,...,lm-l) 

is expressible 8jS cL certain sum of 

(id VJ,_,,/,a(/i, . . . , Im-i) 

and 

(id +^/„«,+i)Vl,,/,+ia(/i, . . . , Im-i)- 
It is yet another result, which manifests the connection of V^^y and the recursion. The 
lemma is Lemma 3 of [2] and we omit its proof here. In order to simplify the statement 
we use the following notation T^^y = {id +Sx,y)Vx,y. 

Lemma 3. Let f{li,l2,h) be a function on 1} with values in C and define 

5((fci, ^2,^3,^4) = 5Z fih,k,h)- 

Then 

TkiMdih, h, h) 

_ / fca ka k4 k2+l ks—l ka-l 

\/l=A;2 + l ^2=^2 + 1 h=k2 h=ki ^2=^2 l3=k2 

^ /k-i-l ka-l ka-l k-i-l 

+ 2 E E Az2(id+^/jT,„,J(/i,/2,fc2)- H Ai,{id+Ei,)Ti,,iJ{k2 + l,l2,h) 

\ll=k2 l2=k2 l2=k2 h=k2 

+ 2 (j^h,l2f{h,l2,k2 + I)l(/i,/2) = (fc2,fc2) ~ Tl^,l2fih,k,h + ^)\(l,^l2) = (k2,k2) 

+ Tl2,hf{k2,l2,h)\(^l^^l^-) = (^k2,k2) ~ ^'2.'3/(^3,/2,/3)|(i2,i3) = (fc2,fc2)^ 

- Tl^,l2f{h,l2,k2+ ^)\{h,l2) = (k2,k3) ~ ^'2,/3/(^2,^2,^3)|(,2,/3) = (fc2,fc3) • 

Moreover, for a function h{li, I2) on 1? , 

{kiMM) ^ fc2-l A:2-l 

Tk^M E Kh^2) = "2 E E Ti,,iMhj2). 

{h,l2) h=ki l2=ki 

This proves the assertion preceding the lemma for m = 3,4. For m = 2 observe that 

(kiM) I k2 

(id+S'fc,,fc2)Vfci,fc2 E ^(^1) ^ (id+5'fci,fc2) I ^ a(^i) - a(A;i) 
(/i) \«i=fci 

A;2 fci 

= ^ a(/i) + ^ a(/i) = 0. 

h=fcl + l i2=fc2 + l 
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In order to use Lemma to prove the assertion for m > 5, we need a merging rule for 
fl2.ip . Let /(x, z) be a function on Z^. Then the operator /^^ is defined as follows. 

IlJix, Z) = fiy -l,y) + fiy, y + I) - f(y - l,y + I) = V.,Jix, ^) I 

Using this operator, we have 

{ki,...,km) {ki,...,ki^2,w) {x,ki,ki+i,y) (z,ki+3,...,lm-i) 

{h,---,lm-l) {h,---,k-2) {h-l,h,li + l) {h + 2 ,■ ■ ■ ,1-n) 

(3.3) 

and this enables one to prove the assertion for m > 5. (For details see 0, Section 4].) 

After noting that {id +Sx,y)g{x,y) = if and only if g{x,y) is antisymmetric in x 
and y, we finally obtain the following. 

Corollary 1. Suppose a(/i, . . . , Im-i) is a function on Z™~^ such that V/.,i^^-^a(/i, . . . , Im-i) 
is antisymmetric in k and /j+i for all i. Then 

(^1 1 ■ ■ ■ ) 

yk„k,+i E '^{h, ■ ■ ■ ,lm-l) 
(ii,...,i,„_i) 

is antisymmetric in ki and ki^i for all i. 

4. Characterizing properties of 7(n; fci, . . . , k^n/2]) 

We apply the results from the previous section to ■y{n, x;ki,..., k^n/2])- Corollary [T] 
implies by induction with respect to n that Vfc-,fc.^^7(n, x; fci, . . . , fcp„/2]) is antisym- 
metric in ki and fcj+i for all i. Lemma [2] and the merging rule (13.31) then implies by 
induction with respect to n that the degree of 7(n, x;ki,..., k^n/2]) in h is no greater 
than n — 1. We summarize this in the following lemma. 

Lemma 4. For n > 1, '~f{n, x;ki, . . . , /^[n/2]) is a polynomial of degree no greater than 
n — 1 in every k^. Furthermore, Vfc.,fc._^i7(n, x; /ci, . . . , k<^n/2\) is antisymmetric in ki and 
ki+i for all i. 

It will be shown that the properties from the previous lemma together with the 
property in the following lemma characterize 7 up to a multiplicative rational constant. 

Lemma 5. If n is even then 

7(n, x; /ci, . . . , /Cn/2-1, kn/2) = -7{n, x;ki,..., kn/2-1, 2x + 2 - kn/2) 
and if n is odd then 

'y{n, x;ki,..., fc(„-i)/2, fc(n+i)/2) = l{n, x;ki,..., k(^n-i)/2, 2x + 1 - k(n+i)/2)- 
In order to prove this lemma, we need another lemma. 
Lemma 6. (1) Let f{li) be such that f{li) = —f{2x + 2 — li) for all li and define 

k2 

g{ki,k2)= J2 /(^i)- Then g{ki,k2) = g{ki,2x + 1 - k2) for all ki,k2. 

h=k\ 

(2) Let fill, I2) he such that f{li, I2) = f{li, 2x + 1 — I2) for all h, I2 and 

(id+^,,,,JV^„zJ(/l,/2)=0. 
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Define 

{ki,k2,x) k2 X 

(/l,/2) Il=kil2=k2 

Then g{ki, ^2) = —g{ki, 2x + 2 — ^2) for all fci, /c2- 
Proof of Lemma\^ (1) By definition, 

2x+l-fc2 k2 2x+l-k2 

gih,2x+l-h)= Yl /(^i) = E /(^i) + E /(^i)- 

(l=fci 'l=A.'l ii=fc2 + l 

2x+l-k2 

The assertion follows since fih) = 0. Tfiis is because 

li=k2+l 

2x+l-k2 2x+l-fc2 22,'+l-fc2 

E /(^i)= E -/(2x+2-/o = - E /(^i)- 

Zl=fc2+1 ;i=fc2+l h=A:2+l 

(2) Observe tliat 

2a;+2-fc2 x 

gih,2x + 2-k2)= E E /(^i,^2)-/(2x + 2-A;2,2x + 2-A;2) 

li=ki l2=2x+2-k2 
k2 X 2x+2—k2 X 

= E E E E /(^i,^2)-/(2x + 2-A;2,2x + 2-A;2) 

Zi=fci ;2=2x+2-fc2 Zi=A:2+l ;2=2x+2-fc2 

k2 X 2x+2-k2 2x+l-k2 2x+2-k2 2x+l-k2 

= E E /(/i,2x+i-/2)-- E E E E 

'l=fci «2=2x+2-A;2 h=k2+l l2=x+l /i=fe2+l l2=x+l 

- f{2x + 2-k2,2x + 2-k2) 

k2 k2-l 2x+2-k2 2x+l-k2 2x+2-k2 2x+l-k2 

= E E E E E E fih,2x+i-k) 

h=ki l2=x+l li=k2+l l2=x+l li=k2+l l2=x+l 

- f{2x + 2-k2,2x + 2-k2) 

k2 X ^ 2x+2-k2 2x+l-k2 , 2x+2-fc2 x 

= -E E/(^i'^^)-2 E E fVuh)-- E E/(^i'^^) 

h=kl l2=k2 li=k2 + l l2=X+l li=k2 + l l2=k2 

- f{2x + 2-k2,2x + 2-k2) 

k2 X ^ 2x+2-k2 2x+l-k2 

= - E E E E /(/i,^2)-/(2x + 2-A;2,2x + 2-fc2). (4.1) 

h=ki l2=k2 li=k2+l l2=k2 

Moreover, we liave {id +Sx,y)Vx,y = {id+EyE~^Sx,y){id+E~^AxAy), and, tliere- 
fore, {id+Ei^E-^Si,,i,){id+E-jAi^Ai^)f{h,l2) =0. Tliis implies tliat 

2x+2-fe2 2x+l-fe2 

^ ^ {id+Ef^'A,Mfik,l2) = 0, 

^1=^2 + 1 ^2=^2 
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since 

2x+2-k2 2x+l-fc2 

2x+2-fc2 2x+l-k2 

= E E -Ei2Ei^'S,,,Uid+K'^iAi.)f{k,l2) 

ll=k2 + l l2=k2 

2x+l~k2 2x+2-k2 

= - E E Si,,iM+Ei-,'/^iAi.)f{hM) 

h=k2 l2=k2+l 

2x+l-k2 2x+2-k2 

= - E E i^d+Ef^'Ai,A,,)fik,k). 

l2=k2 ll=k2 + l 

Thus, (14.1 p is equal to 

k2 x 2x+2-k2 2x+l-k2 

-E E/(^i'^2) + 2 E E E^;'Ai,AiJ{h,k)-f{2x + 2-h,2x + 2-h) 

h=ki l2=k2 h=k2+l l2=k2 

k2 X ^ ^ ^ 

= - E E /(^i' ^2) - 2/(22: + 2-k2,2x + 2-k2) + -f{k2, k2) - -f{2x + 2-k2, k2) 

h=ki l2=k2 

-^f{k2,2x + 2-k2). (4.2) 

Next observe that 

= ((id+5i,,zJV5,,/J(/l,/2))|(,,,;,)=(fe,_l,2x+l-fe) 

= f{k2 -l,2x + l-k2) + f{k2, 2x + 2-k2)~ f{k2 - 1, 2x + 2 - ^2) 

+ f{2x + l-k2,k2-l) + f{2x + 2-k2, k2) - f{2x + 1 - fca, A^a). 

We replace -|/(2x + 2 - k2, k2) - |/(A;2, 2x + 2 - k2) in (|12D by 

]^f{k2-l,2x + l-k2) + ]^f{2x + l-k2,k2-l)-]^f{k2-l,2x + 2-k2)-]^f{2x + l-k2,k2) 

and, consequently, (14.21) is equal to 

k2 X ^ 

"EE /(^i' ^2) + k2) - -if{2x + 2-k2,2x + 2-k2) + fik2, k2) 

h=ki l2=k2 

-f{k2-l,2x + l-k2)-f{2x + l-k2,k2-l) + f{k2-l,2x + 2-k2) + f{2x + l-k2,k2j) 

k2 X ^ 

= - E E /(^i' ^2) + /(^2, k2) - -(/(2x + 2 - A;2, 2x + 2 - ^2) + /(A:2, ^2) 

h=ki l2=k2 

-f{k2-l,k2)-f{2x+l-k2,2x+2-k2)+f{k2-l,k2-l) + f{2x + l-k2,2x+l-k2)). 

(4.3) 
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Finally, 

= ((id +Si,,M,,iJilu ^2))|(,,,,)=(,,,) = 2(/(/, + /(/ + !,/ + !)- /(/, I + 1)) 
implies that fl4.3p is equal to —g{ki, k2). □ 

Proof of Lemma We use induction with respect to n. For n = 2 the assertion is 
easy to check. We assume that n > 3 and first consider the case that n is odd. By the 
recursion, it suffices to show that 

(A:i,...,A;(„_i)/2>'={n+i)/2) 

^ 7(n-l,x;/i,...,/(„_i)/2) 

(il,...,/(„_l)/2) 

(fci,...,fc(„_i)/2.2a;'+l-fc(„+i)/2) 

= X] 7(ra- l,a;;/i,...,/(n_i)/2) (4.4) 

By the induction hypothesis and by Lemma E] (1) we know that 

('=(n-l)/2'''(" + l)/2) (fc'('„_l)/2.2^^ + l~fc{n + l)/2) 

^ 7(n-l,x;/i,...,/(„_i)/2) = ^ 7(n-l,x;/i,...,/(„_i)/2). 

('{n-l)/2) ('(n-l)/2) 

The assertion follows, since the left hand side of (14. 4p is equal to 

('=l'-''=(„-l)/2) (fc'('„-l)/2'fc{n+l)/2) 
(il,...,i(„_3)/2)) {l{n-l)/2) 

and the right hand side of (14.41) is equal to 

(fcl.-.fc'(„_l)/2) (fc'('„_l)/2.2^+l-'=(n + l)/2) 
(il,...,Z(„_3)/2)) ('(n-l)/2) 

In the case that n is even. Lemma [U] (2) is used in a similar way. □ 

5. Derivation of the operator formula 

By Lemma m we know that Vki,ki+i'y{n,x; ki, . . . , /c[n/2]) is antisymmetric in ki and 
ki+i for all i. Although the operators Vki,ki+^ commute, this clearly does not imply that 

Vfci,fc2l42,fc3 • • • Vfcr„/2i-i,fcrn/2i7(«, x; fci, . . . , k^n/2\) 
is antisymmetric in (/ci, k2, ■ . ■ , fc[n/2])- However, it is not hard to see that 

n Vkj„kA lin,x;ki,. . . ,k^n/2-]) (5.1) 

,i<p<9<r"/2i / 

is antisymmetric in {ki, k2, ■ ■ ■ , k^n/2\)- This is a consequence of the following lemma, 
which generalizes [21 Lemma 4]. The proof is analogous to the proof of 0, Lemma 4] 
and thus we omit it here. 
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Lemma 7. Let W^^y be an operator in x and y, which is invertible as operator over 
C[x,y], andW^^^y^W^^^y^ = W^^^y^W^^^y, for all Xi, X2,yi,y2- Moreover, let a{ki, . . . , km) 
be a polynomial in {ki, . . . , km)- Then Wk^^ki+iCtiki, . . . , km) is antisymmetric in ki and 
ki+i for all i if and only if 



is antisymmetric in {ki, . . . , km)- 

We denote the polynomials in flS.ip by 7*(n, x;ki,..., fc[n/2]) and list them for n = 



- 6kix + Akik2X - 6k2X + 12x + kl - kikj + k^ - 6ki - klk2 + 4:kik2 - 6k2 + 11), 
^{k2-ki){k3-ki){k3-k2){8x^-8kix^-8k2x'^-8k3X^ + 12x^+2klx+2klx+2klx-8kix 

+ 6kik2X — 8k2X + Gkik^x + Qk2k^x — 8k^x + 30x + k\ — kik\ + k2 — kik^ — k2kl + kl — lOki 
- klk2 + ?>kik2 - 10A;2 - klk^ - kjks + ?,kik^ - 2kik2k^ + 8/^2/^3 - lOfcg + 25) 

Although this hst is shorter than the analog hst for 7(^,0;; fci, . . . , k\n/2\) (this is due 
to the factor Y\ i^j ~ ^i)i which is a consequence of the antisymmetry of the 



polynomial), it is still hard to guess the general pattern of 7*. Thus we will apply 
a further operator to 7*, in order to obtain a polynomial which factorizes into linear 
factors over Q and for which it is easy to recognize a pattern. This operator will have 
the property that it does not destroy the antisymmetry of the polynomial but restores 
the symmetry property of 7 given in Lemma [5l In the end, the fact that our operators 
are invertible will allow us to "divide" and give a formula for 7 itself. 

The next lemma shows that the application of an operator, which is a symmetric 
polynomial in the shift operators, to an antisymmetric polynomial retains the antisym- 
metry. 

Lemma 8. Let a{ki, . . . , km) be a polynomial that is antisymmetric in {ki, . . . , km) and 
p{Xi, . . . , Xm) be a polynomial in Xi, ^, X2, X2'^, . . . , Xm, X~^, which is symmetric 
in {Xi, . . . , Xm)- Then p^E^j^, . . . , Ek^)a{ki, . . . , km) is antisymmetric in {ki, . . . , km)- 




1,2,3,4,5. 




l<i<j<[n/2] 
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Proof. Let cr e Sm be a permutation and p{Xi, . . . , X^) = Cii,...,im-^l^ ' ' ' -^m- 

{il,...,im) 

The symmetry of p{Xi, . . .,Xm) implies that = Ci^^-^-,,...,i,^^y Thus 

p{Ek„ . . . , Ekja{ki, . . . , k^) = ^ Ci,^,„^i^El\ . . . El^aiki, . . . , km) 

{ii,...,im) 

= Ci,,.,i™a(A;i + «i, . . . , fcm + w) 

= sgn cr ^ Ci^(i),...,v(„)a(/i;a(i) + V(i),---,/i;a(m) +Mm)) 

(jl,...,im) 

m) ''m y 

(il,...,jm) 

= sgn a {p{Ei, Eijaih, /™)) • □ 

In the following lemma we identify operators whose application do not destroy sym- 
metry properties of the type given in Lemma [5l 

Lemma 9. Let a{ki, . . . , km) be a polynomial such that 

(i{ki, . . . , km) = cr ■ ci(^ki, . . . , km—i, d km) 

for a,dEM. andp(Xi, . . . , Xm) be a polynomial in Xi, ^, X2, X2'^, . . . , Xm, X~^ such 
that 

p{X^,...,Xm)=p{X^,...,Xm-l,X-'). 

Set b{ki, . . . , km) = p{Ek^, . . . , Ek^)a{ki, . . . , km) ■ Then we have 

b{ki, ...,km) = cr- b{ki, km-i, d - km) 

as well. 

Proof Let p{Xi, . . . , Xm) = E Qi,...,^™^^ " " " • By assumption Ci,„„,i^_i,i^ = 
Cn,..,i„„i,-i„. Therefore, 

p{Ek^,...,Ek^)a{ki,...,km)= ^ Ci^,...,imaiki + ii, . . . , km + im) 

{ii,...,im) 

= cr ■ ^ ^ Cji,...,im'^(^l "I" ^1) • • • ) km— I + im—li d km im) 
{il,...,im) 

cr ■ ^ ^ Cji,...,im_i,— imCl(/i;i -|- . . . , km— I ~\~ im—li d km "^m) 

(jl,...,im) 

= cr • ^ ^ Cn,...,i,„Q'(^l + ^1; • • • ; + — km + ^m) 

(il,...,im) 

= a ■ {p{Ei„. . . , Eijaih, lm))\^i,_i^)=^k„...,k^.„d-krr.) ■ ° 
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The previous two lemmas suggest to look for an operator p{Ek^, . . . , E^^^^^^), which 
is, on the one hand, symmetric in (fci, . . . , /cp„/2]) and, on the other hand, has the 
property that the composition 

\l<p<q<\n/2-] 

is invariant under the replacement of -Efc|-„/2i by E^^^ This is accomplished in the 
following lemma. 

Lemma 10. The polynomial 

H (id+Efc,Afcj£;,;i(id+E,-iAfcJ 7(n,a;;/ci,...,A:r„/2i) 

,i<p«?<r"/2i / 

is antisymmetric in [ki, . . . , fc[n/2])- Moreover, if n is odd then the polynomial is in- 
variant under the replacement of ki by 2x + 1 — ki and if n is even then the replacement 
of ki by 2x + 2 — ki only changes the sign of the polynomial. 

Proof. By Lemma [3, 

Yl (id+Efe^AfcJ -f{n,x;ki,...,k^n/2]) 

^l<p<q<\n/2-] ) 

is antisymmetric in (fci, . . . , k^n/2'\)- Lemma [H] and the fact that 

n x;\i + x-\x,-i))= n {x;' + x^'-x^'x^') 

is symmetric in (Xi, . . . , X|-„/2]) imply that the expression in the statement of the 
lemma is still antisymmetric in (fci, . . . , A;[„/2])- 

Next observe that the operator in the statement of the lemma is a polynomial in 
the shift operators -E^^, which is invariant under the replacement of E^^^^^] by -^fcj-^/2i' 
Therefore, by Lemma Eland by Lemma [HI the second assertion in lemma follows for i = 
[ri/2] . The assertion for general i follows from the antisymmetry of the polynomial. □ 

The next lemma shows that the previous lemma together with the degree estimation 
(Lemma H]) determines /2]) up to multiplicative constant, which only 

depends on n. 

Lemma 11. Let p{ki, . . . , fc[n/2]) be an antisymmetric polynomial in {ki, . . . , fc[n/2]) 
over C of degree no greater than n — 1 in every ki which is, in the case that n is odd, 
invariant under the replacement of ki by 2x + 1 — ki for every i and, in the case that 
n is even, has the property that the replacement of ki by 2x + 2 — ki only changes the 
sign of the polynomial. Then p{ki, . . . , fc[n/2]) equals 

C- Yl {kj - ki){2x + 1 - ki - kj) 

l<j<j<(n+l)/2 
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if n is odd and 

C \ W {kj-k.i){2x + 2-ki-kj)\\{{x + l-h) 

\l<i<3<n/2 J i=l 

if n is even, where C is a constant in C 

Proof. We only consider the case that n is even for the other case is analogous. A 
polynomial . . . , kn/2) that is antisymmetric in (/ci, . . . , kn/2) must have kj — ki as 
a factor since 

p{ki, . . . , ki—i, ki, /cj+i, • • • , kj-i, ki, kj^i, . . . , kn/2) = 

— p{ki, . . . , ki_i, ki, fcj+i, . . . , kj_i, ki, fcj+i, . . . , kn/2)- 

This is because the polynomial changes the sign if we exchange the element in the i-th 
position with the element in the j-th position. If it furthermore has the property that 
it will change the sign if kj is replaced by 2x + 2 — kj then the polynomial has a zero 
at kj = 2x + 2 — ki which explains the factor 2x + 2 — ki — kj. Moreover it has a zero 
at fcj = X + 1 for every i, since 

p{ki, ki^i, x + l, ki+i, kn/2) = -p{ki, ki_i,x + 1, fcj+i, . . . , kn/2), 

which follows from 2x + 2 — (x + l)=x + l. □ 

Consequently, by Lemma IH Lemma [TU] and Lemma [TH 

n (id+E,^AfeJ^,;(id+E-;AfcJ -1{n,x■,k^,...,k^n/2^) 

l<p<q<\n/2^ ) 

is equal to the polynomials given in Lemma [TTl This determines 7(n, x;ki, . . . , k\n/2'\) 
up to a multiplicative complex constant C„. This is because the operators (id A^^^) 

and {id+E^^^k^ are invertible by Lemma [TJ In the following lemma we compute 

Cn. 

Lemma 12. If n is odd then 

c= n 1 - 1 

J- J- ij-i)ij + i-l) in -l)\in -?,)]■■ -2] 

l<j<j<(n+l)/2 '^-J ' ' ^ ' ^ I 

and if n is even then 

n (j - z)(j + z) n 7 = _ _ . . . ir 

Proof. We expand the polynomial 'jiji, X] ki, . . . , k 

ln/2]) with respect to the basis 

\n/2-] 

n {^i)m^ consider the coefficient of the basis element appearing in this expan- 

i=l 

sion with maximal degree sequence (mi,m2, . . . ,m^n/2'\) in lexicographic order. This 
coefficent is equal to Cn- We show by induction with respect to n that this maximal 
degree sequence is (n — 1, n — 3, . . . , 0) if n is odd and (n — 1, n — 3, . . . , 1) if n is even. 
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An analysis of the definition of Yl induction hypothesis imply that this 

Clvi'm-l) 

maximal basis element of 7(n, x;ki, . . . , /c[n/2]) is the maximal element of 

k2-l ks-l fc(n+l)/2 (n-l)/2 , - 

EE- E n 



J- (n-2i)\ 

Il=kll2=k2 l(n-l)/2=k(„-l)/2 *=1 

if n is odd and the maximal element of 

k2-l ks-l X n/2 . , 

\ ^ -TT [l"i)n-2i 

■ ■ ■ 11 (n _ 2i)\ 

h=kl l2=k2 ln/2=k„/2 «=1 

if n is even. □ 

This immediately implies the following theorem. 
Theorem 2. If n is odd then 

7(n,x;fci,...,fc(„+i)/2) 

n (id+E,^AfcJ-iE,^(id+E,-;AfcJ-i 

^l<p<g<(ri+l)/2 



l<i<j<(n+l)/2 U / 

and if n is even then 
7(n,x; ki,...,kn/2) 

n (id+E,,A,J-iE,^(id+E,7A,J-i 

^l<p<ij<n/2 

n{kj - ki){2x + 2 - ki - kj) X + I - ki 

Proof. The assertion follows from Lemma [U Lemma HI Lemma [TOl Lemma [TT] and 
Lemma [121 □ 

We are finally able to prove the main theorem. 
Proof of Theorem [I]. Observe that 

n (1 + x,{x, - i))(i + x;\x, - i))(i + x,(x;i - i))(i + - 1)) 

l<P<g<rn/2l 
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is invariant under the replacement of Xi by ^. Moreover, it is symmetric in (Xi, . . . , 
since the factor associated to the pair (p, q) is equal to 



,1 + XpXq — Xg) (1 + Xp ^Xg ^ — X^ ^)XpXf^ 



X {I + XpX-' - X-')X, 



X {l+X;^X^-X,)Xp 



and this is symmetric in Xp and Xq as every line is. Thus, by Lemma El 



n (id +EuAk,) (id (id A,J (id -E-^E-^^u,) 

l<p<q<{n+l)/2 

JJ {kj - ki){2x + I - ki - kj) (5.2) 

l<i<j<(n+l)/2 

is antisymmetric in (fci, . . . , k(^n+i)/2) if n is odd and 



n (id+E,,Afcj(id+E,-;A,j(id-Efc,E,;iA,j(id-i5;,7E,-;Afcj 

l<P<'7<"/2 

n/2 

JJ (A;, -A;i)(2x + 2-A;, -A;,)J3(^+1-^*) (5-3) 

is antisymmetric in (fci, . . . , kn/2) if is even. Moreover, by Lemma [HI fl5.2p is invariant 
under the replacement of ki by 2x + 1 — fcj, whereas (15.31) changes the sign if ki is 
replaced by 2x + 2 — k^. Consequently, by Lemma [TTl (15.21) is equal to 

Dn ■ n ~ (2x + 1 - A;, - /cj) 

l<i<i<(n+l)/2 

if n is odd and (15. 3p is equal to 

n/2 

Dn ■ n ~ + 2 - /c, - A;,) J3(a; + I - 

l<«<J<"/2 «=1 
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if n is even. If we compare the coefficient of a monomial of maximal degree we see that 
Dn = 1. Now, if n is odd then, by Theorem [21 

7(n,a;; h, . . . , k(n+i)/2) 

H (id+E,,A,J-iE,^(id+E,;iA,J-i 

J<p<q<(n+l)/2 

n {kj - ki){2x + 1- kj - kj) 

l<i<j<(n+l)/2 \J J\-J I 

n (id+E,,Afej-iE,^(id+E,-iA,j-i 

J<p<g<(n+l)/2 

n (id A,j (id (id A,j (id A,,: 

J<p<g<(n+l)/2 

n (fcj - /Cj)(2x + 1 - fcj - fcj) 

l<j<j<(n+l)/2 ^^-^ ^ 

n i?..(id -i?.,i?,->,J(id -i?,7i?,-;A,^: 
,i<p<g<("+i)/2 



l<i<j<(n+l)/2 ' 

Similarly, if n is even then 
7(n,a;; /ci, . . . , fc„/2) 

n i?.,(id -E, E,-;A,J(id -i?,;ii?,-;A,J 

l<p<ij<n/2 

n/2 

n(A;j /cj) (2x + 2 ki kj) -r-r x + 1 /cj 

l<i<i<n/2 ' y j=l 

6. From operator formulas to generating functions 

In this section we follow a hint of Doron Zeilberger and translate the operator for- 
mulas into generating function results. We start out with ordinary monotone trian- 
gles. In [2] we have shown that the number of monotone triangles with bottom row 
(fci, . . . , fc„) is given by 



a(n; /ci,...,fc„) 



n (id+i?.A-i?,j) n 

\\<p<q<n / l<i<j<n *^ 
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if fci < ^2 < ■ ■ ■ < and fcj e Z. We define adn; ki, . . . , kn) = Yl ~fij' if ^« — ^ 

l<i<j<n 

for all / and zero elsewhere. Then 

a{n; fci, . . . , = I JJ (id +Ek^Ek^ - ^fc J j ac(^; fci, • • • , fcn) 

\l<p<g<n / 

for all (fci, . . . , with ki > c. We compute the generating function 

J2 ^1 ' • • • ^n" ( n (id +^^.^^. - Ek,) ) «o(n; fci, . . . , fc„). 

(A;i,..,fc„)>(-n+l,...,-n.+l) \l<p<g<ra / 

(6.1) 

Thus, the coefficient of X'^'X^^ • • gives the number of monotone triangles with 
bottom row (fci, . . . , kn) if < ki < k2 <...< kn and ki G Z. Let 

n il + Y,Y,-Y,)= J2 aij,,...,jn)Yr---Y^". 

l<p<q<n (ii,.-..jn) 

0<ji<n-l 



Using this notation, fl6.ll) is equal to 



aiji,---,Jn)X^' ■..X'^-ao{n;ki+ji,...,kn + Jn) (6.2) 



(A;i,...,fc„)>(— n+l,...,— n+l) (3i,---,jn) 

0<jj<n-l 



We set (/i, . . . , /„) = {ki + ji, . . . , kn + in)- Consequently, fl6.2p is equal to 



Y a(ji,...,j„)Xi-^\..X->' ao(n;/i,...,gX^...X^. 

(6.3) 



(iiv.'n)>(-n+l+ii,.--,-n+l+in) 

0<jj <n — 1 



Since ao(^i; . . . , /n) = if /j < for an z and ji < n — 1 for all /, (16.31) is equal to 
Y • • • , Jn)X^'' . . . X-^- Y Mn-Ji,..., ln)X[^ ...Xt 

Ol,--.Jn) («l,...,i„)>(0,...,0) 

0<j^ <'n, — 1 

= n E ■'f!'----^-;." n '-0- 

i<p<g<" (/i,...,/„)>(o,...,o) i<j<j<™ 

The Vandermonde determinant evaluation implies that 



-LA j — i i<i,j<n V 7 — 1/ 



l<i<j/<n 

and, consequently, the generating function is equal to 



l<p<q<n {h,...M>(0,...,0) 
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Observe that 

^ -— n - x V^ = 

(j - ' {1-xy 

Therefore, the generating function is equal to 



:i-xi)(i-x2)...(i-x„)^ 



<i<i<n 



n 



(Xj - X.i){l - Xj + XiXj] 



(1 - - X2) ... (1 - x„) ^^11^^ - X,)X,il - X,) 

n 

i=l l<i<j<n 

where the Vandermonde determinant evaluation is used again. We summerize the 
result in the following theorem. 

Theorem 3. The coefficient ofX^'X^'' • • in 

n 

n ^n-in-ir^n n {x,-xi){i-x,+x,x,) (6.4) 

is equal to the number of monotone triangles with bottom row {ki, . . . , kn) if < ki < 
k2 < . . . < kn and ki G Z, where fl6.4p is interpreted as a formal laurent series, 



l-Xi 

j=0 

Consequently, the enumeration of n x n alternating sign matrices amounts to compute 
the constant term of 

n 

U x--^-^a-x)n n (x,-x.)(i-x, + x.x,). 

i=l l<i<j<n 

This is because monotone triangles with bottom row (0, 1, . . . — 1) correspond to 
n X n alternating sign matrices. Zeilberger pjj has used constant term identities to 
give the first proof of the alternating sign matrices theorem. (His identities are different 
from our result.) 

Note that, if, for instant, we choose n = 3 in the generating function in Theorem [3] 
and consider the coefficient of Xj'XiXs we obtain —1, which is oviously not the number 
of monotone triangles with bottom row (3, 2, 1), since there exists no monotone triangle 
with this property. This coefficient is of course the values of a(3;3,2,l). On the 
other hand, if we consider monomials X^^ • • • X^" with negative exponents then their 
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coefficients are not equal to a{n; ki, . . . , kn): for example the coefficient of 

is 7 and this is not a{3; —1, 2, 3) = 23. (In order to compute the number of monotone 

triangles with bottom row (—1, 2, 3) using the generating function from Theorem[3l one 

can make use of the fact that a{3; —1, 2, 3) = a{3; — 1 + c, 2 + c, 3 + c) for all integers 

c.) 



Next we derive an analog generating function for halved monotone triangles with 
prescribed bottom row. Observe that Theorem [T] is equivalent to 

-f{n,x;ku...,kin/2]) = 

n + - + id -E^^') I 7(n, x; fci, . . . , k^^/,^) 

J<p<q<ln/2] 

where 

-TT (kj - ki){2x + 2-n- ki- kj) 
7(n,x;fci,...,fc(„+i)/2)= [[ 

i<j<j<{n+i)/2 ^-^ ''VJ "T ; 

if n is odd and 

^(n T-k k ,\- TT {kj-k,){2x + 4-n-ki-k,) ^ x + 2-n/2-ki 

l<i<j<n/2 '^-^ ' i=l 

if n is even. Here, we define 7c(^, x; fci, . . . , /c|'n/2]) to be equal to 7(n, x; fci, . . . , /i;[n/2]) 
if < c for all / and zero elsewhere. Then 

7(n,x;fci,...,/cp„/2]) = 

^1<P<9<K21 / 

for all (fci, . . . , fc[n/2]) with ki < c. We compute the generating function 

^1 ■■■^\n/2] 

(fci,...,A;p„/2l)<("-l+Cv,n-l+c) 



X 



n + - + id -E,^') I 7c(^, x-k,,..., (6.5) 

l<p<g<[n/2] 



Let 



\n/2] ■ 

-n + l<jj<0 
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Thus, the generating function (16.51) is equal to 

{ki,...,k^„j2])<in-l+c,...,n-l+c) (ii.---Jf„/21 

-n + l<jj<0 

X 7^(n,x; fci + ji, • • • , A;p„/2] + j[n/2-])- (6.6) 

Again we set {h, /[n/21) = {h+ji, k\n/2\ +i\n/2\)- Consequently, ([HIS]) is equal 
to 



/21 



-n+l<jj<0 

X 7>,x;/i,...,/r„/2i)Xl\..X;^%^;. (6.7) 

(ii,...,/|-„/2i)<{"-i+ii+Cv,"-i+irn/2i +c) 

Since j; > — n + 1 for all / and 7c(^, x; /i, . . . , /|'ri/2]) = if > c for an i, fl6.7p is equal 
to 



Y 6(ji,---,jrn/2i)^r^^ 



/21 



-n + l<jj<0 



X XI 7c(^,^;^i,---,^K2i)^l'---^p 



h V r"/2i 
n/21 

(Zi,...,/|-„/2l)<(Cv,c) 



= W (Xp + X,-l)(XpX, + l-X,) 

i<p<g<r'i/2l 

X X 7>,x;/i,...,/r„/2i)X^...x[|:/2^;. (6.8) 

(Zi,...,Zp„/2])<{c,...,c) 

In the last expression 7^ can be replaced by 7. The following lemma provides us with 
determinantal expressions for 7(n, x; fci, . . . , fc|-n/2])- 

Lemma 13. (1) 

det /^^' + ~ -"^^ - TT (fcj - fei)(fc» + kj) -pr fc^ 

l<ij<". I 2? - 1 j J-l (j-{\(jJ^{\ 11 ^ 

'fci + J - 3/2\ _ -p-r (% - fci) {ki + /Cj ) 



l<M<nV 2j-2 ; ^^ll^J^-_^)(^- + ^_l) 

Proof. We only prove (1) since the proof of (2) is similar. First observe that 
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Thus, the determinant in (1) is equal to 

n — 1 n 



n(^n\,tA4n«-'^))- 



i=l ^ ' i=\ \/=l 

The assertion follows from 



det v,m= n (^^-^^ 



l<i<j<n 

where is a polynomial in Y of degree j — 1 whose leading coefficient is 1. This 

is a consequence of the Vandermonde determinant evaluation. □ 

Lemma [T3] implies that 

l<«,j<(n+l)/2 \^ 2j — / y 

if is odd and 

7(n,x;A;i,...,fc„/2) = (-!)( ^ ) det ' \ / (6.10) 



if n is even. If we use these determinantal presentations for 7 in (16. Sp . we obtain the 
following generating function 

J] (l-Xp-X,)(XpX, + l-X,) 

i<p<g<(n+i)/2 

X det + — 5m A 

l<i,i<(n+l)/2 V 2j-2 i ' I ^ ^ 



if n is odd, and 



n (1 - - ^,)(Xp^, + 1 - 

i<p<g<"/2 

+ j + ?^/2 — X — 3 



if n is even. 

If we choose c = x + 1/2 — nj2 in case that n is odd and c = x + 2 — nj2 in case 
that n is even, the determinants in the expression above simplify. This follows from 
the following identities. 



2,-2 ; (2, - 2)! (,^^^ 



(2j - 2)! rfX2i~2 \^ X - 1 y (2j - 2)! rfX2i-2 - ly (X 



T)iFT (6-13) 



where z = —j 
Similarly, 
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— 1, —j, . . . ,j — 3. (Note that the identity is true for all j if z = —2.) 



l=—oo 



where z = —j — 1, —j, . . . ,j — 2. (This identity is true for all j if z = —2, —1.) 

We first consider the case that n is odd. By (16.131) {z = —2) the generating function 
(16. lip is equal to 



n (1 - X, - X,)(X,X, + 1 - X,) det (jx^-. 

l<p<g<(n+l)/2 -''-^ J/ \\ t ) 

(n+l)/2 , , j_l 

= n (1-.V,-A-,)(.W1-A-,) n (xf^,<,,^et , ((X^j 



The Vandermonde determinant evaluation shows that this is equal to 



(n+l)/2 ^^^+1 



X'' 

n {i-x,-x,){x,x,+i-x,) n 



l<P<9<(n+l)/2 

^ n vfx -D^ - IV 

l<i<j<(n+l)/2 ' ^ ' 



Xj 



(n+l)/2 ^■+3/2-n/2 

n (x,-x,)(x,+x,-i)(x,x,-i)(i-x,+x,x,) n ^. _ • 

l<j<j<{n+l)/2 i=l * 

(6.15) 



Finally we consider the case that n is even. By (I6.14p {z = —1) the generating 
function (I6.12p is equal to 



j+c 



n (1 - X, - X,)(X,X, + 1 - A-,) ^^et I 

l<p<q<n/2 - ,J- / \\ I I 



n (1 - X, - A,)(.VY, + 1 - A,) n ,4et.,, ( (X. - ir- 

l<p<q<n/2 i=l ^ ^ > - 'J- I \\ t j . 
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The Vandermonde determinant evaluation now shows that this is equal to 



'^/2 yc+l 






n/2 yX+^~n/2 



= n {X,-X,){X, + X,-l)iX,X,-l){l-X, + X,X,)]Jj^-—. (6.16) 



In this case the generating functions in f l6.15p and f l6.16p are understood as formal 
laurent series in i.e. l/{Xi — 1) = J2jl-oo-^i ■ 
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